Papers by Christopher D Manning
Sneaking Syntax into Transformer Language Models with Tree Regularization (2025.naacl-long)
Copied to clipboard
| Challenge: | Existing methods for incorporating syntactic inductive biases into transformers are limited . we introduce auxiliary loss function that converts bracketing decisions into differentiable orthogonality constraints on vector hidden states. |
| Approach: | They propose to introduce syntactic inductive biases into transformer circuits through a structured regularizer. |
| Outcome: | The proposed approach could unlock more robust and data-efficient learning in transformer language models . it integrates seamlessly with the standard LM objective, requiring no architectural changes. |
Drop Dropout on Single Epoch Language Model Pretraining (2025.findings-acl)
Copied to clipboard
| Challenge: | Initial dropout was seen as a breakthrough regularization technique that reduced overfitting, yet single-epoch pretraining tasks common to modern LLMs yield minimal overfit. |
| Approach: | They propose to use dropout during single-epoch pretraining to reduce overfitting in language modeling, morpho-syntax, question answering, and MNLI to improve performance. |
| Outcome: | The results show that dropout is not used in large LLMs and improves performance in language modeling, morpho-syntax, question answering, and MNLI. |
Stronger Baselines for Retrieval-Augmented Generation with Long-Context Language Models (2025.emnlp-main)
Copied to clipboard
| Challenge: | Existing long-context language models (LMs) can handle tens of thousands of tokens in a single context window. |
| Approach: | They compare two recent multi-stage pipelines, ReadAgent and RAPTOR, against three baselines. |
| Outcome: | The proposed pipelines outperform more complex methods on multiple long-context QA benchmarks. |
Humans and transformer LMs: Abstraction drives language learning (2026.eacl-long)
Copied to clipboard
| Challenge: | lexical semantic and syntactic categories emerge using novel divergence-based metrics . |
| Approach: | They compare transformer-based language model's linguistic categories learning to exemplar-based accounts of human language acquisition. |
| Outcome: | The proposed model can be used as an existence proof for human language acquisition. |
Mechanisms vs. Outcomes: Probing for Syntax Fails to Explain Performance on Targeted Syntactic Evaluations (2025.emnlp-main)
Copied to clipboard
| Challenge: | Existing studies have not evaluated whether probing accuracy predicts syntactic outcomes. |
| Approach: | They evaluate 32 open-weight transformer models and find that probing fails to predict outcomes of targeted syntax evaluations across English linguistic phenomena. |
| Outcome: | The proposed model does not predict syntactic outcomes on English linguistic phenomena. |
LawInstruct: A Resource for Studying Language Model Adaptation to the Legal Domain (2025.findings-naacl)
Copied to clipboard
Joel Niklaus, Lucia Zheng, Arya D. McCarthy, Christopher Hahn, Brian M Rosen, Peter Henderson, Daniel E. Ho, Garrett Honke, Percy Liang, Christopher D Manning
| Challenge: | In general, instruction tuning is important for direct user interaction, but the legal domain is underrepresented in typical instruction datasets. |
| Approach: | They aggregate 58 annotated legal datasets and write instructions for each to create LawInstruct. |
| Outcome: | The proposed model improves on LegalBench across all model sizes, but no drop in MMLU. |